Electronic device, robotic system, and virtual area setting method

ABSTRACT

An electronic device associates coordinates of a virtual space with coordinates of a real space. The electronic device includes an imaging section, a recognition section, a first setting section, and a display controller. The imaging section captures an image or respective images of a hand and a robot in the real space to generate a captured image of the hand and the robot. The recognition section recognizes a gesture represented by a motion of the hand based on the captured image. The first setting section sets the robot in the virtual space based on coordinates of the hand in the virtual space when it is recognized that the gesture corresponds to a first gesture. The display controller controls display of the robot so that the robot is visible to the human eye.

INCORPORATION BY REFERENCE

The present application claims priority under 35 U.S.C. § 119 toJapanese Patent Application No. 2019-97768, filed on May 24, 2019. Thecontents of this application are incorporated herein by reference intheir entirety.

BACKGROUND

The present disclosure relates to an electronic device, a roboticsystem, and a virtual area setting method.

A robot controlling device controls a robot based on a simulationresult. Specifically, the robot controlling device sets an area in avirtual space based on the simulation result. Here, in the area, anoperation of the robot is restricted.

Such a simulation is carried out by a simulation device. The simulationdevice carries out a simulation of an operation of a virtual robot inthe virtual space. The simulation device carrying out the simulationrequires creating a three-dimensional model of a work space where therobot works.

SUMMARY

An electronic device according to an aspect of the present disclosureassociates coordinates of a virtual space with coordinates of a realspace. The electronic device includes an imaging section, a recognitionsection, a first setting section, and a display controller. The imagingsection captures an image or respective images of a subject and a robotin the real space to generate a captured image of the subject and therobot. The recognition section recognizes a gesture represented by amotion of the subject based on the captured image. The first settingsection sets a virtual area in the virtual space based on coordinates ofthe subject in the virtual space when it is recognized that the gesturecorresponds to a first gesture. The display controller controls displayof the virtual area so that the robot is visible to the human eye. Thevirtual area corresponds to a boundary area in which an operation of therobot is restricted in the real space.

A robotic system according to an aspect of the present disclosureincludes an electronic device and a robot control device. The electronicdevice associates coordinates of a virtual space with coordinates of areal space. The robot control device controls a robot. The electronicdevice includes an imaging section, a recognition section, a firstsetting section, and a display controller. The imaging section capturesan image or respective images of a subject and the robot in the realspace to generate a captured image of the subject and the robot. Therecognition section recognizes a gesture represented by a motion of thesubject based on the captured image. The first setting section sets avirtual area in the virtual space based on coordinates of the subject inthe virtual space when it is recognized that the gesture corresponds toa first gesture. The display controller controls display of the virtualarea so that the robot is visible to the human eye. The robot controldevice includes a setting section. The setting section sets, based onthe virtual area, a boundary area in which an operation of the robot isrestricted in the real space.

A virtual area setting method according to an aspect of the presentdisclosure is a method of setting a virtual area in coordinates of avirtual space associated with coordinates of a real space. The virtualarea setting method according to the aspect of the present disclosureincludes: capturing an image or respective images of a subject and arobot in the real space to generate a captured image of the subject andthe robot; recognizing a gesture represented by a motion of the subjectbased on the captured image; setting a virtual area in the virtual spacebased on coordinates of the subject in the virtual space when it isrecognized that the gesture corresponds to a first gesture; andcontrolling display of the virtual area so that the robot is visible tothe human eye. The virtual area corresponds to a boundary area in whichan operation of the robot is restricted in the real space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a robotic system according to a first embodiment ofthe present disclosure.

FIG. 2 illustrates a head-mounted display according to the firstembodiment.

FIG. 3A illustrates an example of a first gesture in the firstembodiment.

FIG. 3B illustrates another example of the first gesture in the firstembodiment.

FIG. 3C illustrates still another example of the first gesture in thefirst embodiment.

FIG. 4 illustrates a second gesture in the first embodiment.

FIG. 5 illustrates a third gesture in the first embodiment.

FIG. 6 is part of a flowchart illustrating an operation of a controllerof the head-mounted display according to the first embodiment.

FIG. 7 is the remaining part of the flowchart illustrating the operationof the controller of the head-mounted display according to the firstembodiment.

FIG. 8 illustrates a robot control device in a second embodiment of thepresent disclosure.

FIG. 9 is a flowchart illustrating an operation of a controller of therobot control device in the second embodiment.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure will hereinafter bedescribed with reference to the accompanying drawings. Elements that arethe same or equivalent are labelled with the same reference signs in thedrawings and description thereof is not repeated.

First Embodiment

A first embodiment of the present disclosure will be described withreference to FIGS. 1 to 7. The first embodiment enables a human toeasily set a virtual area in which an operation of a robot is restrictedusing mixed reality (MR).

A configuration of a robotic system 100 according to the firstembodiment will be described with reference to FIG. 1. FIG. 1illustrates the robotic system 100. As illustrated in FIG. 1, therobotic system 100 includes a head-mounted display 10, a network hub 20,a robot control device 30, and the robot 40. The head-mounted display 10associates coordinates of a virtual space with coordinates of a realspace. The head-mounted display 10 corresponds to one example of an“electronic device”.

In the first embodiment, the head-mounted display 10 is a videosee-through wearable terminal. The head-mounted display 10 is allowed tobe worn on a human head. The head-mounted display 10 displays an imageso that the image is visible to the eye of a wearer. The wearercorresponds to one example of a “human”.

The head-mounted display 10 includes an imaging section 11 and a display14.

The imaging section 11 captures an image of a subject 500 in a field ofview of the wearer of the head-mounted display 10. The imaging section11 captures an image of the robot 40 in the field of view of the wearerof the head-mounted display 10. Specifically, the imaging section 11captures an image or respective images of the subject 500 and the robot40 in the real space to generate a captured image of the subject 500 andthe robot 40. The imaging section 11 includes for example one or morecameras. Examples of the camera(s) include a complementary metal oxidesemiconductor (CMOS) image sensor and a charge coupled device (CCD)image sensor.

In the first embodiment, the subject 500 is a hand of the wearer.Therefore, in the description below, the subject 500 is referred to asthe hand 500 in order to facilitate understanding. The subject 500 mayhowever be for example an object and not limited to the hand.

The display 14 display an image so that the image is visible to the eyeof the wearer. The display 14 includes a display device. Examples of thedisplay device include a liquid-crystal display and an organicelectroluminescent (EL) display.

In the first embodiment, the display 14 displays an image obtained bymixing a captured image generated by the imaging section 11 and anobject(s) in the virtual space. Specifically, the display 14 displays animage obtained by mixing the captured image of the hand 500 and therobot 40 and the virtual area 200 set in the virtual space.

The virtual area 200 is an area in which the operation of the robot 40is restricted. In other words, the virtual area 200 corresponds to aboundary area 300 in which the operation of the robot 40 is restrictedin the real space. Specifically, the boundary area 300 is an arearepresenting a boundary, in the real space, between a work space of therobot 40 and a space in which the operation of the robot 40 isrestricted or prohibited.

Each of shapes of the virtual and boundary areas 200 and 300 may be anyshape and not limited to a plane shape. Each shape of the virtual andboundary areas 200 and 300 may be for example a three-dimensional shape.Examples of the three-dimensional shape include a shape surrounding awhole periphery of the robot 40 (for example square tube shape) and ashape arranged at part of the periphery of the robot 40 (for exampleL-shape).

The virtual area 200 is set in the virtual space by the head-mounteddisplay 10 based on a specific gesture J1 by the hand 500. In the firstembodiment, it is therefore possible to easily set the virtual area 200in which the operation of the robot 40 is restricted. The specificgesture J1 is hereinafter referred to as a “first gesture J1”.

Note that the head-mounted display 10 measures the real space to linkthe coordinates of the real space and the coordinates of the virtualspace together. That is, the head-mounted display 10 enables“geometrical matching (positioning)” between the virtual space and thereal space. Performance examples of the “geometrical matching(positioning)” between the virtual space and the real space may includeperformance based on an image obtained by the imaging section 11(examples thereof include a marker base, a model base, and a naturefeature base), performance based on information acquired by a physicalsensor (examples thereof include a depth meter, an acceleration sensor,a gyro sensor, a magnetic sensor, and an inertial measurement unit), andperformance based on the image obtained by the imaging section 11 andthe information acquired by the physical sensor. The physical sensor ismounted on the head-mounted display 10.

In addition, the head-mounted display 10 converts coordinates of thevirtual area 200 in the virtual space to the coordinates of the realspace and calculates the boundary area 300 corresponding to the virtualarea 200. The head-mounted display 10 then transmits information on theboundary area 300 (for example, coordinates of the boundary area 300 inthe real space) to the robot control device 30 through the network hub20.

The network hub 20 relays communication between the head-mounted display10 and the robot control device 30. The network hub 20 also relayscommunication between the robot 40 and the robot control device 30.

Specifically, the network hub 20 performs wireless communication withthe head-mounted display 10 to transmit and receive data thereto andtherefrom. The type of the wireless communication is not particularlylimited as long as the type of the wireless communication is the same asthat of the head-mounted display 10. Note that the network hub 20 mayperform wired communication with the head-mounted display 10.

In addition, the network hub 20 performs wired communication with therobot control device 30 and the robot 40 to transmit and receive datathereto and therefrom. The type of the wired communication is notparticularly limited as long as data transmission and reception arepossible. Note that the network hub 20 may perform wirelesscommunication with the robot control device 30 and the robot 40.

The robot control device 30 controls the robot 40 through the networkhub 20. Specifically, the robot control device 30 controls respectiverotations of motors each of which is a driving source of the robot 40 inaccordance with an operation pattern of the robot 40. The embodimentincludes operation patterns determined for each of works to be performedby the robot 40. Each of the operation patterns represents a movement ofthe robot 40 according to a corresponding work.

In the first embodiment, the robot control device 30 receivesinformation on the boundary area 300 in the real space from thehead-mounted display 10 through the network hub 20 in particular. Therobot control device 30 then controls the robot 40 based on the boundaryarea 300. Specifically, the robot control device 30 controls theoperation of the robot 40 based on the boundary area 300.

For example, the robot control device 30 controls the robot 40 so thatthe robot 40 does not cross a boundary of the boundary area 300. Forexample, the robot control device 30 controls the robot 40 so that therobot 40 is stopped when crossing the boundary of the boundary area 300.Alternatively, the robot control device 30 controls the robot 40 so thatthe robot 40 decelerates when crossing the boundary of the boundary area300. Note that the robot 40 crossing the boundary of the boundary area300 means the robot 40 crossing the boundary of the boundary area 300 ina direction that the robot 40 leaves an installation position of therobot 40.

For example, in the case where a human or an object crosses the boundaryof the boundary area 300 toward the robot 40, the robot control device30 restricts the operation of the robot 40. Specifically, in the casewhere the human or the object crosses the boundary of the boundary area300 toward the robot 40, the robot control device 30 controls the robot40 so that the robot 40 is deactivated.

The robot 40 operates according to control by the robot control device30. For example, the robot 40 holds an article. For example, the robot40 moves the held article to a predetermined location.

Specifically, the robot 40 includes a base section 41, an arm section42, and a hand section 43. The arm section 42 is arranged on the basesection 41. The arm section 42 is freely movable from the base section41 as a starting point in the real space.

Specifically, the arm section 42 includes joints 42 a and arms 42 b. Thejoints 42 a are attached to the arms 42 b with joints 42 a provided onboth ends of each arm 42. Each of the arms 42 b rotates according to acorresponding joint 42 a being driven. Each of the joints 42 a is drivenby a corresponding built-in motor. Each of the built-in motors iscontrolled by the robot control device 30. Thus, the robot controldevice 30 controls the respective built-in motors of the joints 42 a,thereby controlling an operation of the arm section 42.

The hand section 43 holds the article. The hand section 43 is providedat a distal end of the arm section 42. Specifically, the hand section 43is attached to a joint 42 a at the distal end of the arm section 42. Thehand section 43 includes a first finger 43 a and a second finger 43 b.The hand section 43 opens and closes the first and second fingers 43 aand 43 b, thereby releasing and holding the article. A built-in motor inthe hand section 43 opens and closes the first and second fingers 43 aand 43 b. The built-in motor of the hand section 43 is controlled by therobot control device 30. Thus, the robot control device 30 controls thebuilt-in motor of the hand section 43, thereby controlling an operationof the hand section 43.

Next, the head-mounted display 10 will further be described withreference to FIG. 2. FIG. 2 illustrates the head-mounted display 10. Asillustrated in FIG. 2, the head-mounted display 10 further includes acontroller 12, storage 13, and a wireless communication interface (I/F)15.

The wireless communication interface 15 performs wireless communicationwith the network hub 20. The wireless communication is for exampleshort-range wireless communication. The short-range wirelesscommunication is for example wireless communication with a communicationdistance of several meters to several ten meters. Examples of theshort-range wireless communication include communication using BLUETOOTH(registered Japanese trademark), communication using ZIGBEE (registeredJapanese trademark), and communication using WIFI (registered Japanesetrademark).

The controller 12 controls the imaging section 11, the storage 13, thedisplay 14, and the wireless communication interface 15. The controller12 includes a processor such as a central processing unit (CPU).

The storage 13 includes a storage device and stores therein data and acomputer program. Specifically, examples of the storage 13 include anauxiliary storage device and main memory such as semiconductor memory.Examples of the auxiliary storage device include semiconductor memory, asolid state drive, and a hard disk drive. The storage 13 may includeremovable media.

The controller 12 includes a recognition section 12 a, a first settingsection 12 b, a second setting section 12 c, and a display controller 12d. Specifically, the processor of the controller 12 executes thecomputer program stored in the storage device of the storage 13, therebyfunctioning as the recognition section 12 a, the first setting section12 b, the second setting section 12 c, and the display controller 12 d.

The recognition section 12 a recognizes a gesture represented by amotion of the hand 500 based on the captured image generated by theimaging section 11. Specifically, the recognition section 12 arecognizes an image representing a gesture, contained in the capturedimage. In the case where the captured image contains an imagerepresenting a gesture, the recognition section 12 a recognizes agesture type represented by an image contained in the captured image.Specifically, the recognition section 12 a detects presence or absenceof the first gesture J1 to be contained in the captured image.

In addition, the recognition section 12 a can detect presence or absenceof a second gesture J2 to be contained in the captured image. The secondgesture J2 is different from the first gesture J1. Moreover, therecognition section 12 a can detect presence or absence of a thirdgesture J3 to be contained in the captured image. The third gesture J3is different from the first and second gestures J1 and J2. Further, therecognition section 12 a can detect presence or absence of a fourthgesture J4 to be contained in the captured image. The fourth gesture J4is different from the first to third gestures J1 to J3.

In the case where the recognition section 12 a recognizes that a gestureby the hand 500 corresponds to the first gesture J1, the first settingsection 12 b sets the virtual area 200 in the virtual space based oncoordinates of the hand 500 in the virtual space.

The first embodiment therefore enables the wearer to easily set thevirtual area 200 by making the first gesture J1. Here, in the virtualarea 200, the operation of the robot 40 is restricted. For example, thevirtual area 200 is defined by the coordinates of the virtual area 200in the virtual space. The coordinates of the virtual area 200 in thevirtual space are one example of information on the virtual area 200.

In the case where the recognition section 12 a recognizes that a gestureby the hand 500 corresponds to the second gesture J2, the first settingsection 12 b changes a size of the virtual area 200 based on thecoordinates of the hand 500 in the virtual space. The first embodimenttherefore enables the wearer to easily change the size of the virtualarea 200 in the virtual space by making the second gesture J2.

In the case where the recognition section 12 a recognizes that a gestureby the hand 500 corresponds to the third gesture J3, the first settingsection 12 b changes an orientation of the virtual area 200 based on thecoordinates of the hand 500 in the virtual space. The first embodimenttherefore enables the wearer to easily change the orientation of thevirtual area 200 in the virtual space by making the third gesture J3.

In the case where the recognition section 12 a recognizes that a gestureby the hand 500 corresponds to the fourth gesture J4, the first settingsection 12 b moves the virtual area 200 based on the coordinates of thehand 500 in the virtual space. The first embodiment therefore enablesthe wearer to easily change a location of the virtual area 200 in thevirtual space by making the fourth gesture J4.

The second setting section 12 c converts the coordinates of the virtualarea 200 to the coordinates of the real space. Such coordinateconversion can be used without any particular limitation as long as thecoordinate conversion has been already used in the mixed reality field.

The second setting section 12 c sets the boundary area 300 in the realspace based on the virtual area 200 after the coordinate conversion. Thefirst embodiment therefore controls the robot 40 based on the boundaryarea 300, thereby making it possible to easily restrict the operation ofthe robot 40.

For example, the boundary area 300 is defined by the coordinates of theboundary area 300 in the real space. The coordinates of the boundaryarea 300 in the virtual space are one example of information on theboundary area 300 in the virtual space. The coordinates of the boundaryarea 300 are represented for example by a coordinate system where therobot 40 is placed in the real space.

Specifically, the second setting section 12 c sets the virtual area 200after the coordinate conversion as the boundary area 300 in the realspace. The virtual area 200 after the coordinate conversion isrepresented by the coordinates in the real space.

The second setting section 12 c may however set the boundary area 300based on the virtual area 200 after the coordinate conversion and amaximum movable range of the robot 40. This enables setting of a moreappropriate boundary area 300 according to the maximum movable range ofthe robot 40. Information on the maximum movable range of the robot 40is stored in the storage 13.

In addition, the second setting section 12 c may set, based on thevirtual area 200 after the coordinate conversion and an operationpattern of the robot 40, a boundary area 300 according to an operationpattern, corresponding to the operation pattern of the robot 40, of theoperation patterns. This enables setting of a more appropriate boundaryarea 300 according to the operation pattern of the robot 40. Informationon the operation patterns of the robot 40 is stored in the storage 13.

Note that the second setting section 12 c may set the boundary area 300based on the virtual area 200 after the coordinate conversion, themaximum movable range of the robot 40, and an operation pattern of therobot 40.

The display controller 12 d controls display of the virtual area 200 sothat the robot 40 is visible to the eye of the wearer. The firstembodiment therefore enables the wearer to easily set the virtual area200 by making the first gesture J1 while viewing the robot 40 and thevirtual area 200.

Similarly, the embodiment enables the wearer to easily change the size,the orientation, and the location of the virtual area 200 by making thesecond gesture J2, the third gesture J3, and the fourth gesture J4,respectively, while viewing the robot 40 and the virtual area 200.

In the first embodiment, the display controller 12 d causes the display14 to display the virtual area 200 so that the robot 40 is visible tothe eye of the wearer. Specifically, the display controller 12 dgenerates an image obtained by superposing the virtual area 200 on animage of the robot 40. The display controller 12 d provides the image tothe display 14.

The gestures in the first embodiment will next be described withreference to FIGS. 3A to 5. Each of FIGS. 3A, 3B, and 3C illustrates thefirst gesture J1. The first gesture J1 functions as an instructioncausing the first setting section 12 b to set the virtual area 200 inthe virtual space.

FIG. 3A illustrates the first gesture J1 described with reference toFIG. 1. As illustrated in FIG. 3A, the first gesture J1 is a gesturerepresented by the wearer moving the hand 500 in a first direction D1.

In the example of FIG. 3A, the virtual area 200 is set to a rectangularshape extending in the first direction D1 by the first gesture J1. Notethat although the first direction D1 is substantially parallel to avertical direction in the example of the FIG. 3A, the first direction D1is not limited in particular. For example, the first direction D1 may besubstantially parallel to a horizontal direction.

FIG. 3B illustrates another example of the first gesture J1. Asillustrated in FIG. 3B, the first gesture J1 is a gesture represented bythe wearer moving his or her right and left hands 501 and 502 in asecond direction D2 so that a base h1 between a thumb and a forefingerof the right hand 501 is separated from a base h2 between a thumb and aforefinger of the left hand 502.

In the example of FIG. 3B, the virtual area 200 is set to a rectangularshape whose diagonal ends corresponds to the bases h1 and h2. Note thatalthough the second direction D2 is a diagonal direction going up fromleft to right in the example of FIG. 3B, the second direction D2 is notlimited in particular. For example, the second direction D2 may be adiagonal direction going down from left to right.

FIG. 3C illustrates still another example of the first gesture J1. Asillustrated in FIG. 3C, the first gesture J1 is a gesture represented bythe wearer tapping desired locations in the space with the forefinger ofthe hand 500 to determine corners of the virtual area 200. In theexample of FIG. 3C, the virtual area 200 is set to a rectangular shapeas a result of a point P1, a point P2, and a point P3 being tapped.

FIG. 4 illustrates the second gesture J2. As illustrated in FIG. 4, thesecond gesture J2 is a gesture represented by the wearer pressing a leftside of the virtual area 200 with the left hand 502 and sliding a rightside of the virtual area 200 in a third direction D3 while pressing ordragging the right side of the virtual area 200 with the forefinger ofthe right hand 501. The right side of the virtual area 200 is moved inthe third direction D3 by the second gesture J2, so that the size of thevirtual area 200 is changed. Note that part of the virtual area 200pressed with the forefinger of the right hand 501 is not necessarily anend of the virtual area 200. In this case, the virtual area 200 can beextended to a point where the right hand 501 does not actually reach.

Note that although the third direction D3 is a right direction in theexample of FIG. 4, the third direction D3 is not limited in particular.For example, the third direction D3 may be a left direction. Inaddition, the second gesture J2 may be a gesture represented by thewearer pressing the right side of the virtual area 200 with the righthand 501 and sliding the left side of the virtual area 200 in a thirddirection D3 (left direction) while pressing or dragging the left sideof the virtual area 200 with the forefinger of the left hand 502.

FIG. 5 illustrates the third gesture J3. As illustrated in FIG. 5, thethird gesture J3 is a gesture represented by the wearer opening thethumb and the forefinger of the hand 500 and rotating the fingers in arotational direction R1 in the virtual area 200. The virtual area 200 isrotated according to the third gesture J3 and the orientation of thevirtual area 200 is changed. Note that although the rotational directionR1 is a counterclockwise direction in the example of FIG. 5, therotational direction R1 is not limited in particular. The rotationaldirection R1 may be a clockwise direction.

A virtual area setting method performed by the head-mounted display 10will next be described with reference to FIGS. 6 and 7. In the virtualarea setting method, the virtual area 200 is set in coordinates of thevirtual space associated with the coordinates of the real space.

FIGS. 6 and 7 depict part and the remaining part of a flowchartillustrating an operation of the controller 12 of the head-mounteddisplay 10. That is, FIGS. 6 and 7 illustrate the virtual area settingmethod performed by the controller 12 of the head-mounted display 10. Asillustrated in FIGS. 6 and 7, the virtual area setting method includesSteps S101 to S113. The controller 12 therefore executes each of StepsS101 to S113 in a process depicted in the flowchart.

The operation illustrated in FIGS. 6 and 7 starts when a human wears thehead-mounted display 10 on the head and the head-mounted display 10 isactivated.

As illustrated in FIG. 6, at Step S101, the recognition section 12 a ofthe controller 12 starts obtaining the captured image from the imagingsection 11. That is, the imaging section 11 captures an image orrespective images of the hand 500 and the robot 40 in the real space andgenerates a captured image of the hand 500 and the robot 40. The processthen proceeds to Step S102.

At Step S102, the recognition section 12 a recognizes a gesturerepresented by a motion of the hand 500 based on the captured image. Theprocess then proceeds to Step S103.

At Step S103, the recognition section 12 a determines whether or not thegesture corresponds to the first gesture J1. At Step S103, when it isdetermined that the gesture does not correspond to the first gesture J1(No at Step S103), the process proceeds to Step S102. In contrast, atStep S103, when it is determined that the gesture corresponds to thefirst gesture J1 (Yes at Step S103), the process proceeds to Step S104.

At Step S104, the display controller 12 d of the controller 12 controlsdisplay of the virtual area 200 according to the first gesture J1 sothat the robot 40 is visible to the eye of the wearer. In the firstembodiment, the display controller 12 d causes the display 14 to displaythe virtual area 200 so that the robot 40 is visible to the eye of thewearer. Specifically, the display controller 12 d controls the display14 so that the display 14 displays the robot 40 and the virtual area200. The process then proceeds to Step S105.

At Step S105, the first setting section 12 b of the controller 12 setsthe virtual area 200 in the virtual space based on the coordinates ofthe hand 500 in the virtual space. Specifically, when the first gestureJ1 is stopped, the first setting section 12 b sets the virtual area 200in the virtual space based on the coordinates of the hand 500 in thevirtual space. Note that even at Step S104, the first setting section 12b defines the virtual area 200 based on the coordinates, in the virtualspace, of the hand 500 performing the first gesture J1 (namely, movinghand 500). The process then proceeds to Step S106.

At Step S106, the recognition section 12 a recognizes the gesture basedon the captured image. The process then proceeds to Step S107.

At Step S107, the recognition section 12 a determines whether or not thegesture corresponds to the second gesture J2. At Step S107, when it isdetermined that the gesture does not correspond to the second gesture J2(No at Step S107), the process then proceeds to Step S109 illustrated inFIG. 7. In contrast, at Step S107, when it is determined that thegesture corresponds to the second gesture J2 (Yes at Step S107), theprocess proceeds to Step S108.

At Step S108, the first setting section 12 b changes the side of thevirtual area 200 according to the second gesture J2. The process thenproceeds to Step S109 illustrated in FIG. 7.

As illustrated in FIG. 7, at Step S109, the recognition section 12 arecognizes the gesture based on the captured image. The process thenproceeds to Step S110.

At Step S110, the recognition section 12 a determines whether or not thegesture corresponds to the third gesture J3. At Step S110, when it isdetermined that the gesture does not correspond to the third gesture J3(No at Step S110), the process proceeds to Step S112. In contrast, atStep S110, when it is determined that the gesture corresponds to thethird gesture J3 (Yes at Step S110), the process proceeds to Step S111.

At Step S111, the first setting section 12 b changes the orientation ofthe virtual area 200 according to the third gesture J3. The process thenproceeds to Step S112. At Step S112, the second setting section 12 c ofthe controller 12 converts the coordinates of the virtual area 200 tothe coordinates of the real space and sets the boundary area 300 basedon the virtual area 200 after the coordinate conversion. Note that atStep S112, the second setting section 12 c may set the boundary area 300based on the maximum movable range and/or the operation pattern of therobot 40, and the virtual area 200 after the coordinate conversion. Theprocess then proceeds to Step S113.

At Step S113, the controller 12 transmits information on the boundaryarea 300 to the robot control device 30. The process illustrated inFIGS. 6 and 7 then ends.

As above, the robotic system 100 according to the first embodiment ofthe present disclosure has been described with reference to FIGS. 1 to7. The first embodiment enables the wearer of the head-mounted display10 to easily set the virtual area 200 and the boundary area 300 usingthe mixed reality. Further, the first embodiment need not change theconfiguration of the robot control device 30, thus enabling a reductionin cost for introduction of the robotic system 100.

Second Embodiment

A second embodiment of the present disclosure will be described withreference to FIGS. 1, 8, and 9. The second embodiment enables a human toeasily set a virtual area 200 using mixed reality. Here, in the virtualarea 200, an operation of a robot 10 is restricted.

A subject that sets a boundary area 300 corresponding to the virtualarea 200 in the second embodiment is different from that in the firstembodiment. Specifically, in the first embodiment, the head-mounteddisplay 10 sets the boundary area 300 based on the virtual area 200. Incontrast, in the second embodiment, a robot control device 30 sets theboundary area 300 based on the virtual area 200.

Respective configurations of the first and second embodiments are thesame as each other except that the subjects are different from eachother as described above. Therefore, of constituent elements of thesecond embodiment, identical constituent elements to those of the firstembodiment are not described.

A head-mounted display 10 in the second embodiment sets the virtual area200 and does not set the boundary area 300. The head-mounted display 10in the second embodiment therefore includes no second setting section 12c unlike the first embodiment. Therefore, the head-mounted display 10 inthe second embodiment performs Steps S101 to S111 and does not performStep S112. Also in the second embodiment, at Step S113, the head-mounteddisplay 10 transmits information on the virtual area 200 (for example,coordinates of the virtual area 200 in a virtual space) to the robotcontrol device 30.

FIG. 8 illustrates the robot control device 30 in the second embodiment.As illustrated in FIG. 8, the robot control device 30 includes acontroller 31 and storage 32.

The controller 31 controls the storage 32. The controller 31 includes aprocessor such as a CPU. The storage 32 includes a storage device andstores therein data and a computer program. Specifically, examples ofthe storage 32 include an auxiliary storage device and main memory suchas semiconductor memory. Examples of the auxiliary storage deviceinclude semiconductor memory, a solid state drive, and a hard diskdrive. The storage 32 may include removable media.

The controller 31 includes a setting section 31 a and a robot controller31 b. Specifically, the processor of the controller 31 executes thecomputer program stored in the storage device of the storage 32, therebyfunctioning as the setting section 31 a and the robot controller 31 b.

The setting section 31 a performs processing similar to that of thesecond setting section 12 c illustrated in FIG. 2. That is, the settingsection 31 a sets the boundary area 300 based on the virtual area 200.Here, in the boundary area 300, an operation of the robot 40 isrestricted. Specifically, the setting section 31 a receives informationon the virtual area 200 in the virtual space from the head-mounteddisplay 10. Specifically, the information represents the coordinates ofthe virtual area 200 in the virtual space. The setting section 31 a thenconverts the coordinates of the virtual area 200 in the virtual space tocoordinates of a real space. Such coordinate conversion is performedusing the same method as that used in the second setting section 12 c inthe first embodiment. The setting section 31 a sets the boundary area300 based on the virtual area 200 after the coordinate conversion. Otherprocessing of the setting section 31 a is similar to that of the secondsetting section 12 c illustrated in FIG. 2.

The setting section 31 a may set the boundary area 300 based on amaximum movable range of the robot 40 and the virtual area 200 after thecoordinate conversion. This enables setting of a more appropriateboundary area 300 according to the maximum movable range of the robot40. Information on the maximum movable range of the robot 40 is storedin the storage 32.

Further, the setting section 31 a may set, based on the virtual area 200after the coordinate conversion and an operation pattern of the robot40, a boundary area 300 according to an operation pattern, correspondingto the operation pattern of the robot 40, of operation patterns. Thisenables setting of a more appropriate boundary area 300 according to theoperation pattern of the robot 40. Information on the operation patternsof the robot 40 is stored in the storage 32.

The robot controller 31 b controls the robot 40 through a network hub 20based on the operation patterns stored in the storage 32. In the secondembodiment, the robot controller 31 b restricts the operation of therobot 40 based on the boundary area 300 in particular.

A process by the controller 31 of the robot control device 30 whensetting the boundary area 300 will next be described with reference toFIG. 9. The operation illustrated in FIG. 9 starts when the head-mounteddisplay 10 transmits the information on the virtual area 200.

As illustrated in FIGS. 8 and 9, at Step S201, the setting section 31 aof the controller 31 receives the information on the virtual area 200from the head-mounted display 10. The process then proceeds to StepS202.

At Step S202, the setting section 31 a sets a boundary area 300 based onthe information on the virtual area 200. Specifically, the settingsection 31 a sets the boundary area 300 based on the virtual area 200after the coordinate conversion to the real space. Note that at StepS202, the setting section 31 a may set the boundary area 300 based onthe maximum movable range and/or the operation pattern of the robot 40and the virtual area 200 after the coordinate conversion to the realspace. The process illustrated in FIG. 9 then ends.

As above, the robotic system 100 according to the second embodiment ofthe present disclosure has been described with reference to FIGS. 8 to9. The second embodiment enables a human to easily set the virtual area200 and the boundary area 300 using mixed reality. The second embodimentalso enables a reduction in a load applied to the head-mounted display10 when the robot control device 30 sets the boundary area 300.

As above, the embodiments of the present disclosure have been describedwith reference to the drawings. However, the present disclosure is notlimited to the above-described embodiments and can be practiced invarious ways within the scope without departing from the essence of thepresent disclosure (for example, (1) and (2) described below).Furthermore, the constituent elements disclosed in the above-describedembodiments may be altered as appropriate. For example, some of allconstituent elements illustrated in an embodiment may be added toconstituent elements of another embodiment. Alternatively, some of allconstituent elements illustrated in an embodiment may be removed fromthe embodiment.

The drawings mainly illustrate schematic constituent elements in orderto facilitate understanding of the disclosure, and thickness, length,numbers, intervals or the like of constituent elements illustrated inthe drawings may differ from actual ones thereof in order to facilitatepreparation of the drawings. Further, the configuration of eachconstituent element described in the above embodiments is merely oneexample that does not impose any particular limitations and may bealtered in various ways as long as such alterations do not substantiallydeviate from the effects of the present disclosure.

(1) Although the electronic device is a video see-through head-mounteddisplay 10 in the embodiments of the present disclosure, the presentdisclosure is not limited to this. For example, the electronic devicemay be an optical see-through head-mounted display. The opticalsee-through head-mounted display renders human surroundings visible tothe human eye. In this case, the head-mounted display may project(image) a virtual area 200 onto a human retina or an optical couplingelement such as a half mirror, or may display the virtual area 200 on adisplay element that transmits light, such as a liquid-crystal display.Note that the display 14 illustrated in FIG. 2 is unnecessary in thecase where the virtual area 200 is projected (imaged) on the humanretina. The electronic device may be a terminal device. The terminaldevice is for example a mobile terminal device. The mobile terminaldevice is for example a smartphone. Using the mobile terminal devicesuch as the smartphone enables introduction of the robotic system 100 atlower cost.

(2) Although a motion of a human hand 500 is utilized as a gesture inthe embodiments of the present disclosure, the present disclosure is notlimited to this. For example, a motion of an operation member having apredetermined shape (for example, a pen having a predetermined shape)may be utilized as a gesture. That is, the subject 500 may be theoperation member.

What is claimed is:
 1. An electronic device that associates coordinatesof a virtual space with coordinates of a real space, comprising: animaging section configured to capture an image or respective images of asubject and a robot in the real space to generate a captured image ofthe subject and the robot; a recognition section configured to recognizea gesture represented by a motion of the subject based on the capturedimage; a first setting section configured to set a virtual area in thevirtual space based on coordinates of the subject in the virtual spacewhen it is recognized that the gesture corresponds to a first gesture;and a display controller configured to control display of the virtualarea so that the robot is visible to a human eye, wherein the virtualarea corresponds to a boundary area in which an operation of the robotis restricted in the real space.
 2. The electronic device according toclaim 1, wherein when it is recognized that the gesture corresponds to asecond gesture that is different from the first gesture, the firstsetting section changes a size of the virtual area based on thecoordinates of the subject in the virtual space.
 3. The electronicdevice according to claim 1, wherein when it is recognized that thegesture corresponds to a third gesture that is different from the firstgesture, the first setting section changes an orientation of the virtualarea based on the coordinates of the subject in the virtual space. 4.The electronic device according to claim 1, further comprising a secondsetting section configured to perform coordinate conversion byconverting coordinates of the virtual area to the coordinates of thereal space, and set the boundary area based on the virtual area afterthe coordinate conversion.
 5. The electronic device according to claim4, wherein the second setting section sets the boundary area based onthe virtual area after the coordinate conversion and a maximum movablerange of the robot.
 6. The electronic device according to claim 4,wherein the second setting section sets, based on an operation patternof the robot and the converted virtual area, the boundary area accordingto an operation pattern, corresponding to the operation pattern of therobot, of operation patters.
 7. The electronic device according to claim1, wherein the electronic device is a wearable terminal.
 8. A roboticsystem comprising: an electronic device configured to associatecoordinates of a virtual space with coordinates of a real space; and arobot control device configured to control a robot, wherein theelectronic device includes: an imaging section configured to capture animage or respective images of a subject and a robot in the real space togenerate a captured image of the subject and the robot; a recognitionsection configured to recognize a gesture represented by a motion of thesubject based on the captured image; a first setting section configuredto set a virtual area in the virtual space based on coordinates of thesubject in the virtual space when it is recognized that the gesturecorresponds to a first gesture; and a display controller configured tocontrol display of the virtual area so that the robot is visible to ahuman eye, wherein the robot control device sets, based on the virtualarea, a boundary area in which an operation of the robot is restrictedin the real space.
 9. The robotic system according to claim 8, whereinthe setting section sets the boundary area based on the virtual area anda maximum movable range of the robot.
 10. The robotic system accordingto claim 8, wherein the setting section sets, based on an operationpattern of the robot and the virtual area, the boundary area accordingto an operation pattern, corresponding to the operation pattern of therobot, of operation patters.
 11. A virtual area setting method forsetting a virtual area in coordinates of a virtual space associated withcoordinates of a real space, comprising: capturing an image orrespective images of a subject and a robot in the real space to generatea captured image of the subject and the robot; recognizing a gesturerepresented by a motion of the subject based on the captured image;setting the virtual area in the virtual space based on coordinates ofthe subject in the virtual space when it is recognized that the gesturecorresponds to a first gesture; and controlling display of the virtualarea so that the robot is visible to a human eye, wherein the virtualarea corresponds to a boundary area in which an operation of the robotis restricted in the real space.